Adaptive video background modeling using color and depth

Copyright 2001 IEEE. Published in the 2001 International Conference on Image Processing (ICIP-2001), October 7-10, 2001, Thessaloniki, Greece. Personal use of this material is permitted. Permission to reprint/republish for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works, must be obtained from: Manager, Copyrights and Permissions / IEEE Service Center / 445 Hoes Lane / P.O. Box 1331 / Piscataway, NJ 08855-1331, USA. Telephone: + Intl. 732-562-3966. A new algorithm for background estimation and removal in video sequences obtained with stereo cameras is presented. Per-pixel Gaussian mixtures are used to model recent scene observations in the combined space of depth and luminance-invariant color. These mixture models adapt over time, and are used to build a new model of the background at each time step. This combination in itself is novel, but we also introduce the idea of modulating the learning rate of the background model according to the scene activity level on a per-pixel basis, so that dynamic foreground objects are incorporated into the background more slowly than are static scene changes. Our results show much greater robustness than prior state-of-the-art methods to challenging phenomena such as video displays, non-static background objects, areas of high foreground traffic, and similar color of foreground and background. Our method is also well-suited for use in real-time systems.